Classifying warning messages generated by software developer tools

ABSTRACT

A method for classifying warning messages generated by software developer tools includes receiving a first data set. The first data set includes a first plurality of data entries, where each data entry is associated with a warning message generated based on a first set of software codes, includes indications for a plurality of features, and is associated with one of a plurality of class labels. A second data set is generated by sampling the first data set. Based on the second data set, at least one feature is selected from the plurality of features. A third data set is generated by filtering the second data set with the selected at least one feature. A machine learning classifier is determined based on the third data set. The machine learning classifier is used to classify a second warning message generated based on a second set of software codes to one of the plurality of class labels.

TECHNICAL FIELD

The present disclosure relates to classifying warning messages generatedby software developer tools.

BACKGROUND

In some cases, software developers use software developer tools, such asfault finding tools, to analyze or execute software codes to identifypotential flaws in software programs. The fault finding tools cangenerate warning messages for the identified flaws.

DESCRIPTION OF DRAWINGS

FIG. 1 is a flow diagram illustrating an example process for classifyingwarning messages generated by software developer tools, according to animplementation.

FIG. 2 is a flow diagram illustrating an example process for generatinga data set, according to an implementation.

FIG. 3 is a flow diagram illustrating an example method for classifyingwarning messages generated by software developer tools, according to animplementation.

FIG. 4 is a block diagram of an example computer system used to providecomputational functionalities associated with described algorithms,methods, functions, processes, flows, and procedures, as described inthe instant disclosure, according to an implementation.

Like reference numbers and designations in the various drawings indicatelike elements.

DETAILED DESCRIPTION

Software developer tools, such as fault finding tools (FFTs), canidentify potential flaws in software programs. In some cases, FFTs cangenerate a large number of warning messages for either incorrectlyidentified flaws or unimportant flaws that can be ignored by softwaredevelopers, and only a small number of warning messages are for realflaws that need to be fixed by the software developers. Therefore, thesoftware developers usually have to spend time to triage the warningmessages to determine which warning messages to fix. One reason forgenerating such a large number of unimportant warning messages is thatthe FFTs typically consider the software program code itself withoutconsidering contexts of the software program. Contexts of a softwareprogram can include, for example, a development history of the softwareprogram that indicates whether a particular warning message was fixed orignored by software developers in the past. Contexts of a softwareprogram can also include a programming language used in the softwareprogram (for example, C, C++, or java), an application type of thesoftware program (for example, whether the software program is for amobile app, a database application, or a server application), a size ofthe software program (for example, a number of files or a number oflines in the software program), libraries used in the software program,and other context information consistent with this disclosure.

In some cases, a classifier can be used to learn patterns of ignoredwarning messages and fixed warning messages from the development history(i.e., software developers' actions of fixing or ignoring previouslygenerated warning messages from prior runs of the FFT on previousversions of the software program). After learning the patterns, theclassifier can be used to predict whether to fix or ignore a new warningmessage. In other words, the classifier can classify a warning messageto one of two classes with class labels of “fix” and “ignore.” Theclassifier can be a software tool or a program routine for classifying awarning message. For the classifier to learn the patterns, a data setcan be generated based on the development history. The data set caninclude multiple data entries. Each data entry is associated with aprior warning message and a class label of “fix” or “ignore” indicatingwhether the software developers have fixed or ignored that prior warningmessage. In other words, the class label for each data entry is known.As discussed above, frequently most of the warning messages are ignoredby the software developers and a small number of warning messages arefixed. As a result, the data set has a class imbalance. In other words,a majority of the data entries in the data set are associated with oneclass (i.e., the class of “ignore,” called a majority class), and aminority of data entries are associated with another class (i.e., theclass of “fix,” called a minority class). An imbalanced data set is noteffective for the classifier to learn patterns because the classifierdoes not have enough data samples to learn patterns of the minorityclass. Training the classifier using an imbalanced data set can generatea classifier that tends to classify future warnings into the majorityclass and generates many false negatives (i.e., predict a warning as“ignore” while the warning should be classified as “fix”), affecting arecall rate (i.e., a percentage of correctly predicted “fix” instancesout of a total true “fix” instances). In addition, an imbalanced dataset is not effective for selecting a good set of features to include inthe classifier. Existing approaches of evaluating features (so called“feature-set reduction”) typically rank features useful for identifyingthe majority class over features useful for identifying the minorityclass.

At a high level, the described approach generates an accurate classifierusing machine learning methods for an imbalanced data set. Forovercoming the imbalanced data set issue, the described approachaugments the imbalanced data set by sampling the data set so that thesampled data set is approximately balanced or a difference between thenumbers of data entries in a majority class and in a minority class isbelow a threshold. In some implementations, if the classifier does notprovide a good performance, a feedback mechanism can be used tore-sample the imbalanced data set by a different sampling method toadjust learning for the classifier.

In some implementations, from a development history of a softwareprogram, information of various features of the software programassociated with warning messages can be collected to form a data set. Asdiscussed above, the data set can be imbalanced and include multipledata entries. Each data entry is associated with a warning message and aclass label of “fix” or “ignore.” Each data entry can include values ofa number of features of the software program associated with thecorresponding warning message. As will be discussed more below, forexample, a warning message is related to a code error on a specific lineof a specific file in the software program. The features included in thedata entry can be a number of lines of that specific file, a number oflines added or deleted in a most recent code change in that specificfile, and any other suitable information about that specific file. Otherexample features in the data entry are illustrated in Table 1 below.

For generating an accurate classifier, a subset of features thatprovides relevant information towards the warning message classificationare selected. To enable selecting a good set of features, the imbalanceddata set is sampled to generate an approximately balanced data set. Thebalanced data set is used for feature selection. Different from existingmachine learning methods which sample the data set after the featureselection, the described approach sample the imbalanced data set beforethe feature selection.

After the feature selection, the balanced data set is filtered byremoving unselected features. The filtered data set is then divided intotwo sets, a training data set and a testing data set. The training dataset is used to generate and train a classifier to learn the patterns offixed warning messages and ignored warning messages, while the testingdata set is used to evaluate a performance of the classifier. If theclassifier does not provide a good performance, a feedback mechanism isused so that the original imbalanced data set is re-sampled using adifferent sampling method. The re-sampled data set is then used forselecting possibly a different subset of features, and the classifier isre-trained and re-tested. In some implementations, if the feedbackmechanism is not used, no testing data set is needed for evaluating theclassifier performance, and the entire filtered data set can be used asthe training data set.

After learning the patterns of fixed warning messages and ignoredwarning messages, the classifier can be used to classify a new warningmessage. For example, when the software program is further developedwith new codes, a new warning message can be generated by applying theFFT on the further developed software program. Values of the selectedfeatures associated with the new warning messages are collected. Thecollected feature values are provided to the classifier, and theclassifier outputs a class label for the new warning message. If theclassifier classifies the warning message as “ignore,” the warningmessage may not be presented to the software developers. In some cases,the output class labels from the classifier can serve as inputs to asoftware tool, for example, a filter to filter out the “ignore”messages.

In some cases, a new software program may not have sufficient dataentries for a classifier to learn patterns of fixed warning messages andignored warning messages because of a small number warning messagesgenerated during a short software development history. In these cases,data sets from other software programs can be merged with a data setfrom the new software program. The merged data set can be used togenerate a classifier for the new software program. In someimplementations, for each software program to be similarly representedin the merged data set, a normalization can be performed so that a sizeof a large data set is reduced to a similar size of a small data set.

The described approach can also be applied to a classifier and a dataset of more than two classes. For example, data entries in a data setcan be associated with more than two classes. If a ratio of a number ofdata entries in one class to a number of data entries in another classis more than a threshold, the data set can be considered to beimbalanced. Sampling methods can be used to sample the data set so thateach class has a similar number of data entries.

In operation, a first data set can be received. The first data setincludes a first plurality of data entries, where each data entry isassociated with a warning message generated based on a first set ofsoftware codes. Each data entry also includes indications for aplurality of features, and is associated with one of a plurality ofclass labels. A second data set can be generated by sampling the firstdata set. Based on the second data set, at least one feature can beselected from the plurality of features. A third data set can begenerated by filtering the second data set with the selected at leastone feature. A machine learning classifier can be automaticallydetermined based on the third data set. The machine learning classifiercan be used to classify a second warning message generated based on asecond set of software codes to one of the plurality of class labels,where the second set of software codes is different than the first setof software codes. The plurality of class labels can include a firstclass label for fixing the warning message and a second class label forignoring the warning message. The first data set can be an imbalanceddata set. The plurality of features can include features associated withat least one of a software development process, a programming code, asoftware code change, or a fault finding tool analysis. The at least onefeature can be selected based on a cut-off value. The first data set canbe sampled using at least one of over-sampling a minority class orunder-sampling a majority class. The third data set can be divided intoa training data set and a testing data set, where the training data setis used to determine the machine learning classifier, and the testingdata set is used to evaluate a performance of the machine learningclassifier. A stratified sampling or a stratified K-fold sampling can beapplied to the training data set. The machine learning classifier can bea weighted random forest classifier

In some implementations, the first data set can be sampled using a firstsampling method. In response to determining that the performance of themachine learning classifier is lower than a threshold, a differentsecond data set can be generated by sampling the first data set using adifferent sampling method. Different at least one feature can beselected from the plurality of features based on the different seconddata set. A different third data set can be generated by filtering thedifferent second data set with the different selected feature. Adifferent machine learning classifier can be determined based on thedifferent third data set. The second warning message can be classifiedto one of the plurality of class labels using the different machinelearning classifier. In some implementations, a second plurality of dataentries generated based on a third set of software codes can bereceived, where the second plurality of data entries have more dataentries than the first plurality of data entries. The second pluralityof data entries can be modified by removing a set of data entries. Themodified second plurality of data entries can be included in the firstdata set.

The classification of warning messages described herein can use asoftware tool or a program routine to automatically classify a largenumber of warning messages generated by FFTs, and reduce a number ofwarning messages presented to software developers. The describedapproach can generate an accurate classifier to classify warningmessages using an imbalanced data set. For example, the describedapproach can improve the classifier's performance metrics for animbalanced data set that has a size ratio of 9:1 for a majority classand a minority class, in particular improving a performance metric ofrecall for the class of “fix.” In addition, the described approach canalso generate an accurate classifier for a new software program thatdoes not have enough data entries to learn patterns of fixed warningmessages and ignored warning messages. FIGS. 1-4 and associateddescriptions provide additional details for these implementations.

FIG. 1 is a flow diagram illustrating an example process 100 forclassifying warning messages generated by software developer tools,according to an implementation. For clarity of presentation, thedescription that follows generally describes process 100 in the contextof the other figures in this description. However, it will be understoodthat process 100 may be performed, for example, by any suitable system,environment, software, and hardware, or a combination of systems,environments, software, and hardware, as appropriate. For example,process 100 can be performed by the example computer system 400illustrated in FIG. 4. In some implementations, some steps of process100 can be run in parallel, in combination, in loops, or in any order.

At 102, a data set can be generated from a development history of asoftware program. The data set can include multiple data entries. Eachdata entry is associated with a warning message and a class label. Aswill be discussed below, each data entry includes values of a number offeatures associated with the corresponding warning message. In someimplementations, the data set can be in a table format. Each row in thetable is a single data entry. Each column indicates a feature of thedata entry.

In some implementations, historical versions of software program codescan be obtained from a revision control tool. The FFT can be applied onthe historical versions of the software program to generate a list ofwarning messages. For each warning message, a class label is determinedbased on whether the software developers had fixed or ignored thatwarning message. In addition, as discussed in FIG. 2, values of variousfeatures related to that warning message can be collected, for example,by a revision control tool and an FFT. The data set can include featuresfrom, but not limited to, a number of categories such as processfeatures, code features, churn features, and FFT features as shown inTable 1.

TABLE 1 Features in a data set Description Examples ProcessCharacteristics drawn from Number of commits features softwaredevelopment process Number of neighbor's commits Number of normalizedlines Code Characteristics drawn from Number of classes featuressoftware itself Number of files Number of lines Churn Characteristicsdrawn from Number of lines added features recent changes of softwareNumber of lines deleted FFT Characteristics drawn from Severity featuresan FFT analysis Line number

For example, assume that a warning message indicates an error on aspecific line in a particular code file. As shown in Table 1, theprocess features can capture characteristics from the softwaredevelopment process, for instance, a number of commits in the particularcode file. The code features can capture characteristics from thesoftware code itself, for example, a number of lines in the particularcode file. The churn features can capture characteristics of recentchanges of the software code, for example, a number of code lines addedor deleted in the particular code file in a most recent code changeprior to the warning message was generated. The FFT features can capturecharacteristic from the FFT analysis, for example, a line number and anerror severity indicated in the warning message.

As another example, assume that the data set includes the churn featuresand the FFT features shown in Table 1. As a result, the data set can bea table having four columns with column names of “churn feature: numberof lines added,” “churn feature: number of lines deleted,” “FFT feature:severity,” and “FFT feature: line number.” Assume that a particularwarning message indicates a fatal error on line 10 of a specific codefile, and that in a most recent code change prior to the warning messagezero lines were added and three lines were deleted from the specificcode file. In that case, the data entry for that particular warningmessage includes feature values of 0, 3, “fatal,” and 10 for the fourfeature columns. In addition, a class label is associated with that dataentry based on whether the software developers had fixed the particularwarning message.

FIG. 2 is a flow diagram illustrating an example process 200 forgenerating a data set, according to an implementation. For clarity ofpresentation, the description that follows generally describes process200 in the context of the other figures in this description. However, itwill be understood that process 200 may be performed, for example, byany suitable system, environment, software, and hardware, or acombination of systems, environments, software, and hardware, asappropriate. In some implementations, various steps of process 200 canbe run in parallel, in combination, in loops, or in any order.

At 202, 204, 206, and 208, values of process features, code features,churn features, and FFT features, are respectively collected for eachwarning message by, for example, a revision control tool or an FFT. At210, the collected feature values are merged into, for example, a tableformat. At 212, the merged feature values are transformed to a data set.For example, the transformation can convert non-number feature values(such as the error severity feature discussed above whose feature valuecan be a character string) to numbers so that a classifier canmanipulate easily. In some cases, a mapping from one set of codes(non-number) to another set of codes (numbers) can be achieved by asimple fixed function.

Turning back to FIG. 1, in some cases, different software programs canhave different contexts. For example, a flaw that needs to be fixed inone software program can be ignored in another software program.However, in some cases, different software programs can have similarcontexts, that is, the software developers treat the flaws similarly inthose software programs. Two software programs can have similar contextsif, for example, the two software programs are written in similarprogramming languages, have similar application types (for instance,both software programs are for mobile apps), have similar sizes, or usesimilar libraries.

In some instances, a new software program may not have enough dataentries for a classifier to learn patterns of fixed warning messages andignored warning messages because of a small number warning messagesgenerated during the short software development history. At 104, datasets collected from other software programs that have similar contextscan be merged with the data set from the new software program so thatthe merged data set can have a decent size to generate a classifier forthe new software program. In other words, old learning histories fromother software programs can be used for the new software program whosedata set size is not large enough to generate a classifier for itself.If a software program has enough data entries for generating aclassifier, then step 104 is not performed.

In some implementations, when multiple data sets of different sizes aremerged, a normalization can be performed. The number of data entries ina data set is considered to be the size of the data set. Normalizationreduces the size of a big data set to a similar size of a small dataset, for example, by removing some data entries from the big data set.For instance, if the big data set is an imbalanced data set, some dataentries from the majority class can be removed. In some implementations,a normalization tolerance level can be set to ignore the sizediscrepancy up to a predefined level. For example, a normalizationtolerance level of 5% can be set. That is, no normalization is performedif two data sets differ by 5% or less in size. The normalized data setsare merged together to build a new data set. Normalizing multiple datasets to a similar size can reduce a sampling bias. A sampling bias canbe presented if in the merged data set one software program hassignificantly more data entries than another software program.

As discussed above, in some cases, the data set can have a classimbalance. That is, the data entries in the data set are heavily biasedto one specific class label. For example, most of the data entries(e.g., 90%) in the data set are associated with the class label“ignore,” while a small number of data entries (e.g., 10%) areassociated with the class label “fix.” In some cases, a data set isconsidered to have a class imbalance if a ratio of the number of dataentries associated with one class label to the number of data entriesassociated with another class label exceeds a threshold. The classimbalance makes learning ineffective for the classifier because of thesmall pool of data entries for the classifier to learn patterns of theminority class.

At 106, a sampling is performed to reduce the imbalance in the data setso that the different classes in the data set have a similar number ofdata entries to enable an effective learning for the classifier. As willbe discussed below, the described approach can have a feedback loop. Forexample, an initial sampling method can be used to determine an initialclassifier. If the initial classifier does not provide a goodperformance, a different sampling method can be used to determine adifferent classifier that may have a better performance. Varioussampling methods can be used, for example, over-sampling the minorityclass, under-sampling the majority class, and a combination ofover-sampling the minority class and under-sampling the majority class.

A number of under-sampling methods can be used, for example, randommajority under-sampling, under-sampling with cluster centroids, andextraction of majority-minority Tomek links. For example, theunder-sampling with cluster centroids method under-samples the majorityclass by replacing a cluster of majority data entries with the clustercentroid (middle of a cluster) of a k-means algorithm. The randommajority under-sampling method under-samples the majority class byrandomly picking samples with or without replacement.

A number of over-sampling methods can also be used, for example, randomminority over-sampling with replacement, synthetic minorityover-sampling technique, and adaptive synthetic sampling approach. Abasic under-sampling method can simply remove some data entries, while abasic over-sampling method can simply repeat some data entries.Different from existing approaches, the described approach samples theimbalanced data set before feature prediction and selection as describedin 108 and 110, providing a better feature selection to enable aneffective classifier learning. Existing approaches typically applysampling after feature prediction and selection.

As discussed in Table 1, the data set generated in 102 can include alarge number of features. In some cases, a subset of features providesrelevant information to the classification problem of the classifier,while other features provide irrelevant or redundant information.Therefore, for enabling an effective learning for the classifier, asubset of features relevant to the classification problem can beselected, while the irrelevant or redundant features can be removed. Forimproving a performance and reducing a complexity of the classifier,usually a small number of features that capture information relevant tothe classification problem are selected.

At 108, feature prediction can be performed to determine features thatare not correlated to learning patterns of fixed warning messages andignored warning messages so that these features can be excluded from theclassifier. In some implementations, a dendrogram algorithm called avariable clustering can be used along with a variable importance usingboth Mean Decrease Gini and Mean Decrease Accuracy, and a correlationmatrix. In some implementations, a developer tool can read in thesoftware program and generates a list of metrics for the code (e.g.,number of lines in a particular program element). The metrics are mappedto the lines of the program being analyzed. If there is a warningassociated with line L of program P, and the developer tool determinesthat at line L of program P the value for the metric “lines of code inthe program element” is V, then V is the prediction feature. In someimplementations, defect prediction measures can be calculated based onthe development history, such as number of lines of change in theprogram location in a predetermined time period (e.g., the last threemonths). Typically, program parts that are under constant change mayhave more defects. If a warning is issued by the FFT associated withthose rapidly-changing parts, it might be more likely to be a validwarning compared to a different warning that is associated with an areaof code that does not change quickly.

At 110, features that are relevant to the classification problem areselected. In some implementations, a variable clustering method can beused, where one distinct and representative feature is determined fromeach cluster based on a cut-off Spearman value. The cut-off Spearmanvalue can be predetermined by a system operator or an algorithmicprocess. In the case where a single feature is clustered (e.g., in thedendrogram) above the cut-off value, that feature is selected. In someother cases, multiple features can be clustered together either above orbelow the cut-off value. For the clusters below the cut-off value, thecorresponding features are not selected because these features areconsidered not useful enough. For the clusters above the cut-off value,a single feature is selected (e.g., randomly selected) from eachcluster. In some cases, a manual investigation can be made to choose topfeatures based on the cut-off Spearman value. After feature selection,the data set is filtered with the selected features. In other words, amodified data set is formed by removing the table columns of unselectedfeatures from the data set.

The modified data set is used to determine a classifier. For generatingan accurate classifier and mitigating noise effect, a number ofiterations can be used, for example, 10 iterations. In each iteration,the modified data set is split into a training data set and a testingdata set. As will be discussed in 112 and 114, the training data set isused to determine the classifier, while the testing data set is used toevaluate a performance of the determined classifier. In some cases, theclassifier obtained from a previous iteration can be further refined bythe training data set and the testing data set of a current iteration.

In some implementations, the numbers of data entries in the trainingdata set and in the testing data set can have a ratio, e.g., a ratio of3:1. For example, data entries in the modified data set having the classlabel “fix” can be divided into two groups, denoted as Group 1 and Group2, where the numbers of data entries in Group 1 and in Group 2 have aratio of 3:1. Similarly, data entries in the modified data set havingthe class label “ignore” can also be divided into two groups, denoted asGroup 3 and Group 4, where the numbers of data entries in Group 3 and inGroup 4 have a ratio of 3:1. Group 1 and Group 3 are merged to form thetraining data set, while Group 2 and Group 4 are merged to form thetesting data set.

Various sampling methods can be used to split the modified data set andgenerate a balanced training data set and a balanced testing data set(i.e., the data set has a similar number of data entries for eachclass). For example, a stratified sampling method or a stratified-k-foldsampling method can be used for splitting the modified data set. Theidea of the stratified sampling method is to use a probabilisticsampling to divide an entire population into different strata which arerandomly chosen but balanced and proportional. The stratified-k-foldsampling method is different from the stratified sampling method becausethe stratified-k-fold sampling method generates more than a single foldof the stratified sampling.

At 112, the classifier uses the training data set to learn the patternsof fixed warning messages and ignored warning messages. The data entriesof the training data set (i.e., values of selected features) and theassociated class labels are provided to the classifier so that theclassifier can learn patterns of mapping from the feature values to theclass label. The classifier can use Random Forest Classification (RFC)or any other machine learning algorithms. In some cases, samplingmethods discussed in 106 can be used if the training data set is animbalanced data set. Various RFCs can be used, for example, regular RFC,balanced RFC, and weighted RFC. In some implementations, a weighted RFCcan be used where different weights are applied for different classes.In some cases, the weights can be configured by a user or an algorithmicprocess depending on characteristics of the data set.

As discussed above, in some cases, data sets from other softwareprograms can be used so that a new software program can have enough dataentries for the classifier to learn patterns. Different combinations ofthe data sets from multiple software programs can be used to form thetraining data set and the testing data set. For example, data sets fromthree software programs, say X, Y, and Z, are used for the machinelearning. Table 2 shows various methods to form the training data setand the testing data set, where X|Y indicates that the data set ofsoftware program X is used in training and the data set of softwareprogram Y is used in testing.

TABLE 2 Training data set and testing data set for multiple softwareprograms Homogenous training and testing X|X, Y|Y, Z|Z Heterogeneousaugmented testing X|XY, X|XZ Heterogeneous augmented training XY|X, XZ|XHeterogeneous training and testing X|YZ Heterogeneous augmented trainingXYZ|XY, XYZ|YZ, XYZ|XY, and testing XY|XY, XY|YZ

For example, in the case of homogenous training and testing, thetraining data set and the testing data set are considered to behomogenous because both data sets are constructed similarly by includingdata entries from each software program. For instance, the data set ofsoftware program X can be split into two groups, denoted as GroupX1 andGroupX2, where the ratio of the number of entries in GroupX1 to thenumber of entries in GroupX2 is, e.g., 3:1. Similarly, the softwareprogram Y can be split into two groups, denoted as GroupY1 and GroupY2,with a ratio of 3:1, and the software program Z can be split into twogroups, denoted as GroupZ1 and GroupZ2, also with a ratio of 3:1. As aresult, the training data set is formed by merging GroupX1, GroupY1, andGroupZ1, while the testing data set is formed by merging GroupX2,GroupY2, and GroupZ2. As another example, in the case of heterogeneoustraining and testing, the training data set and the testing data set areconsidered to be heterogeneous because the two data sets are formedusing data sets of different software programs. For example, thetraining data set includes the data set from software program X, whilethe testing data set includes the data sets from software programs Y andZ.

Optionally, at 114, the classifier may be tested using the testing dataset to evaluate the performance of the classifier. For each data entryin the testing data set, feature values in the data entry are providedto the classifier, and the classifier outputs a predicted class label(i.e., “fix” or “ignore”). Because the class label associated with eachdata entry in the testing data set is known, the predicted class labelis compared with the known class label to evaluate the classifier'sperformance. Various performance metrics can be collected, e.g.,classification accuracy, precision and recall, and F-measure. Othermetrics, such as confusion matrix, Cohen's Kappa, and a ReceiverOperating Characteristic (ROC) curve, can also be determined.

For example, the classification accuracy can be defined as a percentageof correct predictions. For instance, assume that the testing data sethas 100 data entries, and that the classifier correctly predicted theclass label for 80 data entries. In that case, the classificationaccuracy is 80%. The performance metric of recall can be defined as apercentage of true positives among relevant instances. For example, thetrue positive can be the instance in which the classifier correctlypredicted the class label “fix” (i.e., the predicted class label is“fix” and the known class label is also “fix”). The recall can be apercentage of true positives among the instances in which the knownclass label is “fix.” In other words, the recall measures the percentageof correctly predicted “fix” instances out of a total of true “fix”instances. For example, assume that the testing data set has 100 dataentries, and that 50 out of the 100 data entries are labelled as “fix.”After running the classifier against the testing data set, theclassifier predicted 40 data entries as “fix,” where 30 “fix” labelswere correctly predicted (i.e., the classifier predicted “fix” and thedata entry is also labelled as “fix”), while the other 10 “fix” labelswere incorrectly predicted (i.e., the classifier predicted “fix” but thedata entry is actually labelled as “ignore”). In this case, the recallis considered to be 30/50=60%.

In some implementations, as discussed above, multiple iterations areused for classifier training and testing. The performance metrics canindicate an overall performance from the multiple iterations. Forexample, the recall metric can be calculated for each iteration, then anoverall recall can be calculated by averaging the recalls from themultiple iterations. In some cases, sampling methods discussed in 106can be used if the testing data set is an imbalance data set. In casesof multiple software programs, the testing data set can be formed, forexample, according to Table 2.

After the classifier is tested against the testing data set, theperformance metrics are analyzed. If the performance metrics do notprovide an acceptable result (an acceptable result can be defined bythresholds associated with the performance metrics,), then the process100 goes back to 106 to re-sample the data set generated at 102 (if 104is skipped) or 104 to create a new sampled data set using a differentsampling method (as discussed in 106, a number of different samplingmethods can be used to mitigate the class imbalance). The new sampleddata set is used for feature prediction, feature selection, classifiertraining, and classifier testing as described in 108-112. For example,the process 100 first used the method of over-sampling the minorityclass at 106, if at 114 the classifier does not provide an acceptableresult (for example, the recall is lower than a threshold), the process100 goes back to 106 and selects a different sampling method, e.g.,under-sampling the majority class. The feedback process can continueuntil a sampling method is found that provides an acceptable result.

In some implementations, the feedback process is not used and 114 isoptional. In other words, no testing data set is needed for evaluatingthe classifier performance, and the entire modified data set from 110 isused as the training data set.

After the classifier has been trained to learn the patterns of fixedwarning messages and ignored warning messages, the classifier can beused to classify a new warning message. For example, when the softwareprogram is further developed with new codes, the FFT can generate a newwarning message for the further developed software program. Values ofthe selected features (features selected at 110) for the new warningmessage are determined, for example, using the revision control tool orthe FFT. The collected feature values are provided to the classifier,and the classifier outputs a class label for the new warning message. Ifthe classifier classifies the warning message as “ignore,” the warningmessage may not be presented to the software developers.

FIG. 3 is a flow diagram illustrating an example method 300 forclassifying warning messages generated by software developer tools,according to an implementation. For clarity of presentation, thedescription that follows generally describes method 300 in the contextof the other figures in this description. However, it will be understoodthat method 300 may be performed, for example, by any suitable system,environment, software, and hardware, or a combination of systems,environments, software, and hardware, as appropriate. For example,method 300 can be performed by the example computer system 400illustrated in FIG. 4. In some implementations, various steps of method300 can be run in parallel, in combination, in loops, or in any order.

At 302, a first data set can be received that includes a first pluralityof data entries, where each data entry is associated with a warningmessage generated based on a first set of software codes. Each dataentry includes indications for a plurality of features, and each dataentry is associated with one of a plurality of class labels. Forexample, the first data set can be generated as described in 102 andFIG. 2. At 304, a second data set can be generated by sampling the firstdata set, as described in 106. At 306, based on the second data set, oneor more features can be selected from the plurality of features, asdescribed in 108 and 110. At 308, a third data set can be generated byfiltering the second data set with the selected one or more features.For example, the third data set can be generated by removing featurevalues of unselected features. At 310, a machine learning classifier canbe determined based on the third data set, as described in 112 and 114.At 312, the machine learning classifier is used to classify a secondwarning message generated based on a second set of software codes to oneof the plurality of class labels, where the second set of software codesis different than the first set of software codes.

FIG. 4 is a block diagram of an example computer system 400 used toprovide computational functionalities associated with describedalgorithms, methods, functions, processes, flows, and procedures, asdescribed in the instant disclosure, according to an implementation. Theillustrated computer 402 is intended to encompass any computing devicesuch as a server, desktop computer, laptop/notebook computer, wirelessdata port, smart phone, personal data assistant (PDA), tablet computingdevice, one or more processors within these devices, or any othersuitable processing device, including physical or virtual instances (orboth) of the computing device. Additionally, the computer 402 maycomprise a computer that includes an input device, such as a keypad,keyboard, touch screen, or other device that can accept userinformation, and an output device that conveys information associatedwith the operation of the computer 402, including digital data, visual,or audio information (or a combination of information), or agraphical-type user interface (UI) (or GUI).

The computer 402 can serve in a role as a client, network component, aserver, a database or other persistency, or any other component (or acombination of roles) of a computer system for performing the subjectmatter described in the instant disclosure. The illustrated computer 402is communicably coupled with a network 430. In some implementations, oneor more components of the computer 402 may be configured to operatewithin environments, including cloud-computing-based, local, global, orother environment (or a combination of environments).

At a high level, the computer 402 is an electronic computing deviceoperable to receive, transmit, process, store, or manage data andinformation associated with the described subject matter. According tosome implementations, the computer 402 may also include or becommunicably coupled with an application server, e-mail server, webserver, caching server, streaming data server, or other server (or acombination of servers).

The computer 402 can receive requests over network 430 from a clientapplication (for example, executing on another computer 402) and respondto the received requests by processing the received requests using anappropriate software application(s). In addition, requests may also besent to the computer 402 from internal users (for example, from acommand console or by other appropriate access method), external orthird-parties, other automated applications, as well as any otherappropriate entities, individuals, systems, or computers.

Each of the components of the computer 402 can communicate using asystem bus 403. In some implementations, any or all of the components ofthe computer 402, hardware or software (or a combination of bothhardware and software), may interface with each other or the interface404 (or a combination of both), over the system bus 403 using anapplication programming interface (API) 412 or a service layer 413 (or acombination of the API 412 and service layer 413). The API 412 mayinclude specifications for routines, data structures, and objectclasses. The API 412 may be either computer-language independent ordependent and refer to a complete interface, a single function, or evena set of APIs. The service layer 413 provides software services to thecomputer 402 or other components (whether or not illustrated) that arecommunicably coupled to the computer 402. The functionality of thecomputer 402 may be accessible for all service consumers using thisservice layer. Software services, such as those provided by the servicelayer 413, provide reusable, defined functionalities through a definedinterface. For example, the interface may be software written in JAVA,C++, or other suitable language providing data in extensible markuplanguage (XML) format or other suitable format. While illustrated as anintegrated component of the computer 402, alternative implementationsmay illustrate the API 412 or the service layer 413 as stand-alonecomponents in relation to other components of the computer 402 or othercomponents (whether or not illustrated) that are communicably coupled tothe computer 402. Moreover, any or all parts of the API 412 or theservice layer 413 may be implemented as child or sub-modules of anothersoftware module, enterprise application, or hardware module withoutdeparting from the scope of this disclosure.

The computer 402 includes an interface 404. Although illustrated as asingle interface 404 in FIG. 4, two or more interfaces 404 may be usedaccording to particular needs, desires, or particular implementations ofthe computer 402. The interface 404 is used by the computer 402 forcommunicating with other systems that are connected to the network 430(whether illustrated or not) in a distributed environment. Generally,the interface 404 comprises logic encoded in software or hardware (or acombination of software and hardware) and is operable to communicatewith the network 430. More specifically, the interface 404 may comprisesoftware supporting one or more communication protocols associated withcommunications such that the network 430 or interface's hardware isoperable to communicate physical signals within and outside of theillustrated computer 402.

The computer 402 includes a processor 405. Although illustrated as asingle processor 405 in FIG. 4, two or more processors may be usedaccording to particular needs, desires, or particular implementations ofthe computer 402. Generally, the processor 405 executes instructions andmanipulates data to perform the operations of the computer 402 and anyalgorithms, methods, functions, processes, flows, and procedures asdescribed in the instant disclosure.

The computer 402 also includes a database 406 that can hold data for thecomputer 402 or other components (or a combination of both) that can beconnected to the network 430 (whether illustrated or not). For example,database 406 can be an in-memory, conventional, or other type ofdatabase storing data consistent with this disclosure. In someimplementations, database 406 can be a combination of two or moredifferent database types (for example, a hybrid in-memory andconventional database) according to particular needs, desires, orparticular implementations of the computer 402 and the describedfunctionality. Although illustrated as a single database 406 in FIG. 4,two or more databases (of the same or combination of types) can be usedaccording to particular needs, desires, or particular implementations ofthe computer 402 and the described functionality. While database 406 isillustrated as an integral component of the computer 402, in alternativeimplementations, database 406 can be external to the computer 402.

The computer 402 also includes a memory 407 that can hold data for thecomputer 402 or other components (or a combination of both) that can beconnected to the network 430 (whether illustrated or not). Memory 407can store any data consistent with this disclosure. In someimplementations, memory 407 can be a combination of two or moredifferent types of memory (for example, a combination of semiconductorand magnetic storage) according to particular needs, desires, orparticular implementations of the computer 402 and the describedfunctionality. Although illustrated as a single memory 407 in FIG. 4,two or more memories 407 (of the same or combination of types) can beused according to particular needs, desires, or particularimplementations of the computer 402 and the described functionality.While memory 407 is illustrated as an integral component of the computer402, in alternative implementations, memory 407 can be external to thecomputer 402.

The application 408 is an algorithmic software engine providingfunctionality according to particular needs, desires, or particularimplementations of the computer 402, particularly with respect tofunctionality described in this disclosure. For example, application 408can serve as one or more components, modules, or applications. Further,although illustrated as a single application 408, the application 408may be implemented as multiple applications 408 on the computer 402. Inaddition, although illustrated as integral to the computer 402, inalternative implementations, the application 408 can be external to thecomputer 402.

The computer 402 can also include a power supply 414. The power supply414 can include a rechargeable or non-rechargeable battery that can beconfigured to be either user- or non-user-replaceable. In someimplementations, the power supply 414 can include power-conversion ormanagement circuits (including recharging, standby, or other powermanagement functionality). In some implementations, the power-supply 414can include a power plug to allow the computer 402 to be plugged into awall socket or other power source to, for example, power the computer402 or recharge a rechargeable battery.

There may be any number of computers 402 associated with, or externalto, a computer system containing computer 402, each computer 402communicating over network 430. Further, the term “client,” “user,” andother appropriate terminology may be used interchangeably, asappropriate, without departing from the scope of this disclosure.Moreover, this disclosure contemplates that many users may use onecomputer 402, or that one user may use multiple computers 402.

Implementations of the subject matter and the functional operationsdescribed in this specification can be implemented in digital electroniccircuitry, in tangibly embodied computer software or firmware, incomputer hardware, including the structures disclosed in thisspecification and their structural equivalents, or in combinations ofone or more of them. Software implementations of the described subjectmatter can be implemented as one or more computer programs, that is, oneor more modules of computer program instructions encoded on a tangible,non-transitory, computer-readable computer-storage medium for executionby, or to control the operation of, data processing apparatus.Alternatively, or additionally, the program instructions can be encodedin/on an artificially generated propagated signal, for example, amachine-generated electrical, optical, or electromagnetic signal that isgenerated to encode information for transmission to suitable receiverapparatus for execution by a data processing apparatus. Thecomputer-storage medium can be a machine-readable storage device, amachine-readable storage substrate, a random or serial access memorydevice, or a combination of computer-storage mediums.

The term “real-time,” “real time,” “realtime,” “real (fast) time (RFT),”“near(ly) real-time (NRT),” “quasi real-time,” or similar terms (asunderstood by one of ordinary skill in the art), means that an actionand a response are temporally proximate such that an individualperceives the action and the response occurring substantiallysimultaneously. For example, the time difference for a response todisplay (or for an initiation of a display) of data following theindividual's action to access the data may be less than 1 ms, less than1 sec., or less than 5 secs. While the requested data need not bedisplayed (or initiated for display) instantaneously, it is displayed(or initiated for display) without any intentional delay, taking intoaccount processing limitations of a described computing system and timerequired to, for example, gather, accurately measure, analyze, process,store, or transmit the data.

The terms “data processing apparatus,” “computer,” or “electroniccomputer device” (or equivalent as understood by one of ordinary skillin the art) refer to data processing hardware and encompass all kinds ofapparatus, devices, and machines for processing data, including by wayof example, a programmable processor, a computer, or multiple processorsor computers. The apparatus can also be, or further include specialpurpose logic circuitry, for example, a central processing unit (CPU),an FPGA (field programmable gate array), or an ASIC(application-specific integrated circuit). In some implementations, thedata processing apparatus or special purpose logic circuitry (or acombination of the data processing apparatus or special purpose logiccircuitry) may be hardware- or software-based (or a combination of bothhardware- and software-based). The apparatus can optionally include codethat creates an execution environment for computer programs, forexample, code that constitutes processor firmware, a protocol stack, adatabase management system, an operating system, or a combination ofexecution environments. The present disclosure contemplates the use ofdata processing apparatuses with or without conventional operatingsystems, for example LINUX, UNIX, WINDOWS, MAC OS, ANDROID, IOS, or anyother suitable conventional operating system.

A computer program, which may also be referred to or described as aprogram, software, a software application, a module, a software module,a script, or code can be written in any form of programming language,including compiled or interpreted languages, or declarative orprocedural languages, and it can be deployed in any form, including as astand-alone program or as a module, component, subroutine, or other unitsuitable for use in a computing environment. A computer program may, butneed not, correspond to a file in a file system. A program can be storedin a portion of a file that holds other programs or data, for example,one or more scripts stored in a markup language document, in a singlefile dedicated to the program in question, or in multiple coordinatedfiles, for example, files that store one or more modules, sub-programs,or portions of code. A computer program can be deployed to be executedon one computer or on multiple computers that are located at one site ordistributed across multiple sites and interconnected by a communicationnetwork. While portions of the programs illustrated in the variousfigures are shown as individual modules that implement the variousfeatures and functionality through various objects, methods, or otherprocesses, the programs may instead include a number of sub-modules,third-party services, components, libraries, and such, as appropriate.Conversely, the features and functionality of various components can becombined into single components, as appropriate. Thresholds used to makecomputational determinations can be statically, dynamically, or bothstatically and dynamically determined.

The methods, processes, or logic flows described in this specificationcan be performed by one or more programmable computers executing one ormore computer programs to perform functions by operating on input dataand generating output. The methods, processes, or logic flows can alsobe performed by, and apparatus can also be implemented as, specialpurpose logic circuitry, for example, a CPU, an FPGA, or an ASIC.

Computers suitable for the execution of a computer program can be basedon general or special purpose microprocessors, both, or any other kindof CPU. Generally, a CPU will receive instructions and data from andwrite to a memory. The essential elements of a computer are a CPU, forperforming or executing instructions, and one or more memory devices forstoring instructions and data. Generally, a computer will also include,or be operatively coupled to, receive data from or transfer data to, orboth, one or more mass storage devices for storing data, for example,magnetic, magneto-optical disks, or optical disks. However, a computerneed not have such devices. Moreover, a computer can be embedded inanother device, for example, a mobile telephone, a personal digitalassistant (PDA), a mobile audio or video player, a game console, aglobal positioning system (GPS) receiver, or a portable storage device,for example, a universal serial bus (USB) flash drive, to name just afew.

Computer-readable media (transitory or non-transitory, as appropriate)suitable for storing computer program instructions and data includes allforms of permanent/non-permanent or volatile/non-volatile memory, mediaand memory devices, including by way of example semiconductor memorydevices, for example, random access memory (RAM), read-only memory(ROM), phase change memory (PRAM), static random access memory (SRAM),dynamic random access memory (DRAM), erasable programmable read-onlymemory (EPROM), electrically erasable programmable read-only memory(EEPROM), and flash memory devices; magnetic devices, for example, tape,cartridges, cassettes, internal/removable disks; magneto-optical disks;and optical memory devices, for example, digital video disc (DVD),CD-ROM, DVD+/−R, DVD-RAM, DVD-ROM, HD-DVD, and BLURAY, and other opticalmemory technologies. The memory may store various objects or data,including caches, classes, frameworks, applications, modules, backupdata, jobs, web pages, web page templates, data structures, databasetables, repositories storing dynamic information, and any otherappropriate information including any parameters, variables, algorithms,instructions, rules, constraints, or references thereto. Additionally,the memory may include any other appropriate data, such as logs,policies, security or access data, reporting files, as well as others.The processor and the memory can be supplemented by, or incorporated in,special purpose logic circuitry.

To provide for interaction with a user, implementations of the subjectmatter described in this specification can be implemented on a computerhaving a display device, for example, a CRT (cathode ray tube), LCD(liquid crystal display), LED (Light Emitting Diode), or plasma monitor,for displaying information to the user and a keyboard and a pointingdevice, for example, a mouse, trackball, or trackpad by which the usercan provide input to the computer. Input may also be provided to thecomputer using a touchscreen, such as a tablet computer surface withpressure sensitivity, a multi-touch screen using capacitive or electricsensing, or other type of touchscreen. Other kinds of devices can beused to provide for interaction with a user as well; for example,feedback provided to the user can be any form of sensory feedback, forexample, visual feedback, auditory feedback, or tactile feedback; andinput from the user can be received in any form, including acoustic,speech, or tactile input. In addition, a computer can interact with auser by sending documents to and receiving documents from a device thatis used by the user; for example, by sending web pages to a web browseron a user's client device in response to requests received from the webbrowser.

The term “graphical user interface,” or “GUI,” may be used in thesingular or the plural to describe one or more graphical user interfacesand each of the displays of a particular graphical user interface.Therefore, a GUI may represent any graphical user interface, includingbut not limited to, a web browser, a touch screen, or a command lineinterface (CLI) that processes information and efficiently presents theinformation results to the user. In general, a GUI may include aplurality of user interface (UI) elements, some or all associated with aweb browser, such as interactive fields, pull-down lists, and buttons.These and other UI elements may be related to or represent the functionsof the web browser.

Implementations of the subject matter described in this specificationcan be implemented in a computing system that includes a back-endcomponent, for example, as a data server, or that includes a middlewarecomponent, for example, an application server, or that includes afront-end component, for example, a client computer having a graphicaluser interface or a Web browser through which a user can interact withan implementation of the subject matter described in this specification,or any combination of one or more such back-end, middleware, orfront-end components. The components of the system can be interconnectedby any form or medium of wireline or wireless digital data communication(or a combination of data communication), for example, a communicationnetwork. Examples of communication networks include a local area network(LAN), a radio access network (RAN), a metropolitan area network (MAN),a wide area network (WAN), Worldwide Interoperability for MicrowaveAccess (WIMAX), a wireless local area network (WLAN) using, for example,802.11 a/b/g/n or 802.20 (or a combination of 802.11x and 802.20 orother protocols consistent with this disclosure), all or a portion ofthe Internet, or any other communication system or systems at one ormore locations (or a combination of communication networks). The networkmay communicate with, for example, Internet Protocol (IP) packets, FrameRelay frames, Asynchronous Transfer Mode (ATM) cells, voice, video,data, or other suitable information (or a combination of communicationtypes) between network addresses.

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other.

While this specification contains many specific implementation details,these should not be construed as limitations on the scope of anyinvention or on the scope of what may be claimed, but rather asdescriptions of features that may be specific to particularimplementations of particular inventions. Certain features that aredescribed in this specification in the context of separateimplementations can also be implemented, in combination, in a singleimplementation. Conversely, various features that are described in thecontext of a single implementation can also be implemented in multipleimplementations, separately, or in any suitable sub-combination.Moreover, although previously described features may be described asacting in certain combinations and even initially claimed as such, oneor more features from a claimed combination can, in some cases, beexcised from the combination, and the claimed combination may bedirected to a sub-combination or variation of a sub-combination.

Particular implementations of the subject matter have been described.Other implementations, alterations, and permutations of the describedimplementations are within the scope of the following claims as will beapparent to those skilled in the art. While operations are depicted inthe drawings or claims in a particular order, this should not beunderstood as requiring that such operations be performed in theparticular order shown or in sequential order, or that all illustratedoperations be performed (some operations may be considered optional), toachieve desirable results. In certain circumstances, multitasking orparallel processing (or a combination of multitasking and parallelprocessing) may be advantageous and performed as deemed appropriate.

Moreover, the separation or integration of various system modules andcomponents in the previously described implementations should not beunderstood as requiring such separation or integration in allimplementations, and it should be understood that the described programcomponents and systems can generally be integrated together in a singlesoftware product or packaged into multiple software products.

Accordingly, the previously described example implementations do notdefine or constrain this disclosure. Other changes, substitutions, andalterations are also possible without departing from the spirit andscope of this disclosure.

Furthermore, any claimed implementation is considered to be applicableto at least a computer-implemented method; a non-transitory,computer-readable medium storing computer-readable instructions toperform the computer-implemented method; and a computer systemcomprising a computer memory interoperably coupled with a hardwareprocessor configured to perform the computer-implemented method or theinstructions stored on the non-transitory, computer-readable medium.

What is claimed is:
 1. A method, comprising: receiving, by a hardwareprocessor, a first data set, the first data set including a firstplurality of data entries, wherein each data entry is associated with awarning message generated based on a first set of software codes, eachdata entry includes indications for a plurality of features, and eachdata entry is associated with one of a plurality of class labels;generating, by the hardware processor, a second data set by sampling thefirst data set; based on the second data set, selecting, by the hardwareprocessor, at least one feature from the plurality of features;generating, by the hardware processor, a third data set by filtering thesecond data set with the selected at least one feature; determining, bythe hardware processor, a machine learning classifier based on the thirddata set, wherein determining the machine learning classifier comprisesdividing the third data set into a training data set and a testing dataset, the training data set is used to determine the machine learningclassifier, and the testing data set is used to evaluate a performanceof the machine learning classifier; and classifying, by the hardwareprocessor, a second warning message generated based on a second set ofsoftware codes to one of the plurality of class labels using the machinelearning classifier, wherein the second set of software codes isdifferent than the first set of software codes.
 2. The method of claim1, wherein the plurality of class labels includes a first class labelfor fixing the warning message and a second class label for ignoring thewarning message.
 3. The method of claim 1, wherein the first data set isan imbalanced data set.
 4. The method of claim 1, wherein the pluralityof features includes features associated with at least one of a softwaredevelopment process, a programming code, a software code change, or afault finding tool analysis.
 5. The method of claim 1, wherein samplingthe first data set includes at least one of over-sampling a minorityclass or under-sampling a majority class.
 6. The method of claim 1,wherein at least one of a stratified sampling or a stratified K-foldsampling is applied to the training data set.
 7. The method of claim 1,wherein the first data set is sampled using a sampling method, and themethod further comprises: in response to determining that theperformance of the machine learning classifier is lower than athreshold, generating a different second data set by sampling the firstdata set using a different sampling method; selecting different at leastone feature from the plurality of features based on the different seconddata set; generating a different third data set by filtering thedifferent second data set with the different at least one feature;determining a different machine learning classifier based on thedifferent third data set; and classifying the second warning message toone of the plurality of class labels using the different machinelearning classifier.
 8. The method of claim 1, wherein the machinelearning classifier is a weighted random forest classifier.
 9. Themethod of claim 1, further comprising: receiving a second plurality ofdata entries generated based on a third set of software codes, whereinthe second plurality of data entries have more data entries than thefirst plurality of data entries; modifying the second plurality of dataentries by removing a set of data entries; and including the modifiedsecond plurality of data entries in the first data set.
 10. The methodof claim 1, wherein the at least one feature is selected based on acut-off value.
 11. A device, comprising: a memory; and at least onehardware processor communicatively coupled with the memory andconfigured to: receive a first data set, the first data set including afirst plurality of data entries, wherein each data entry is associatedwith a warning message generated based on a first set of software codes,each data entry includes indications for a plurality of features, andeach data entry is associated with one of a plurality of class labels;generate a second data set by sampling the first data set; based on thesecond data set, select at least one feature from the plurality offeatures; generate a third data set by filtering the second data setwith the selected at least one feature; determine a machine learningclassifier based on the third data set, wherein determining the machinelearning classifier comprises dividing the third data set into atraining data set and a testing data set, the training data set is usedto determine the machine learning classifier, and the testing data setis used to evaluate a performance of the machine learning classifier;and classify a second warning message generated based on a second set ofsoftware codes to one of the plurality of class labels using the machinelearning classifier, wherein the second set of software codes isdifferent than the first set of software codes.
 12. The device of claim11, wherein the first data set is an imbalanced data set.
 13. The deviceof claim 11, wherein sampling the first data set includes at least oneof over-sampling a minority class or under-sampling a majority class.14. The device of claim 11, wherein the first data set is sampled usinga sampling method, and the at least one hardware processor is furtherconfigured to: in response to determining that the performance of themachine learning classifier is lower than a threshold, generate adifferent second data set by sampling the first data set using adifferent sampling method; select different at least one feature fromthe plurality of features based on the different second data set;generate a different third data set by filtering the different seconddata set with the different at least one feature; determine a differentmachine learning classifier based on the different third data set; andclassify the second warning message to one of the plurality of classlabels using the different machine learning classifier.
 15. Acomputer-readable medium containing instructions which, when executed,cause a computing device to perform operations comprising: receiving afirst data set, the first data set including a first plurality of dataentries, wherein each data entry is associated with a warning messagegenerated based on a first set of software codes, each data entryincludes indications for a plurality of features, and each data entry isassociated with one of a plurality of class labels; generating a seconddata set by sampling the first data set; based on the second data set,selecting at least one feature from the plurality of features;generating a third data set by filtering the second data set with theselected at least one feature; determining a machine learning classifierbased on the third data set, wherein determining the machine learningclassifier comprises dividing the third data set into a training dataset and a testing data set, the training data set is used to determinethe machine learning classifier, and the testing data set is used toevaluate a performance of the machine learning classifier; andclassifying a second warning message generated based on a second set ofsoftware codes to one of the plurality of class labels using the machinelearning classifier, wherein the second set of software codes isdifferent than the first set of software codes.
 16. Thecomputer-readable medium of claim 15, wherein the first data set is animbalanced data set.
 17. The computer-readable medium of claim 15,wherein sampling the first data set includes at least one ofover-sampling a minority class or under-sampling a majority class.